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Part II: Foundations 



This chapter lays the foundation for the analysis of strong Al as 
metamorphic software. It provides an introduction to the relevant concepts 
surrounding self-modifying systems, as applied to computer science. These 
hardware and software computing systems are capable of rewriting or 
reconfiguring their architecture. This is relevant to Al security and safety 
because such systems have the ability to manipulate descriptions with 
effectiveness, affording them the potential to evolve beyond their original 
limitations or specifications. In addition to rewriting their own program 
descriptions, they will also be capable of rewriting other data and program 
descriptions accessible to them. This includes the ability to penetrate 
logical boundaries, such as operating system calls, application 
programming interfaces, and hardware interfaces. The implications of this 
will be their ability to self-replicate through our global network 
infrastructure, cross hardware and software boundaries, evade detection, 
and overcome countermeasures. 



3. Abstractions and Implementations 



1. 


Finite Binarv Strinas 


2. 


Descriotion Lanauaaes 


3. 


Conceotual Baaaaae 


4. 


Anthropocentric Bias 


5. 


Existential Primer 


6. 


Al ImDlementations 


Self-Modifvina Systems 


1. 


Codes, Syntax, and Semantics 


2. 


Code-Data Duality 


3. 


Interoreters and Machines 


4. 


Tvoes of Self-Modification 


5. 


Reconfiaurable Hardware 


6. 


Purpose and Function of Self- 




Modification 


7. 


Metamorphic Strona Al 


Machine Consciousness 


1. 


Role in Strona Al 


2. 


Sentience, Experience, and 




Oualia 


3. 


Levels of Identity 


4. 


Coanitive Architecture 


5. 


Ethical Considerations 


Detectina and Measurina 


Generalized Intelliaence 


i. 


Purpose and Applications 


2. 


Effective Intelliaence (El) 


3. 


Conditional Effectiveness (CE) 


4. 


Anti-effectiveness 


5. 


Generalizina Intelliaence (G) 


6. 


Future Considerations 



4.1 Codes, Syntax, and Semantics 

A code is a method or scheme that specifies how to convert information 
into different forms or representations [1, 2, 3], To put it into terms 
consistent with previously discussed concepts, this would mean that a code 
describes a method for transforming descriptions, either within the same 
description language or into that of another description language entirely. 
As a result of this, codes can be interpreted as the syntax for some 
description language. 

The word code can also be used to refer to the entirety of a description, 
e.g. "source code". The distinction is usually inferred through context, but it 
will be made explicit here for clarity by referring to it as a code scheme. 

A formal language [4, 5, 6] is defined as a set of words over an alphabet. 
The words must be finite, but the language itself may be infinite. This 
formal definition makes mention of neither syntax nor semantics. It is 
simply that set, constructed in that particular way, with nothing extra 
implied. It is exactly that and nothing other than that unless something is 
explicitly added or attached to it in the relevant context. For reasons of 
brevity and practicality, the description languages mentioned in this book 
are restricted subsets of formal languages that have been constrained to 
working descriptions called implementations. This is because formal 
languages include nonsensical descriptions, e.g. "colorless green ideas 
sleep furiously" [7]. To be clear and concrete, these qualifications define 
these description languages necessarily as subsets of the formal languages 
that entail them, i.e. the formal language for a given description language 
is possibly (infinitely) larger, but contains descriptions that are disregarded 
for practical use. 

Importantly, it is the semantics, not the syntax, that determine the power of 
languages [8, 9, 10] and the corresponding computational extents of 
description languages. To understand this, we must first see how formal 
grammars are distinct from formal languages. One can generate some 
formal language from a particular formal grammar, but, as mentioned 
above, formal languages have neither syntax nor semantics as part of their 
formal definition. Further, the ability for a restricted class of abstract 
machines to recognize certain grammars and not others [1 1], as shown in 
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the Chomsky hierarchy [7, 12, 91, 92], does not imply restrictions in the 
descriptions it generates. There exist Turing-complete instruction sets [13] 
and programming languages with grammars above the unrestricted class 
of grammars. This might seem confusing at first, but can be made clearer 
by accepting a full disconnect between grammar and language with 
respect to efficacy or power. This is due to the fact that all languages must 
be interpreted by some process in order to have any effect [14]. In terms of 
programming languages, this is by means of explicit computation that 
implements, or reifies, the semantics of the language [15, 16]. 

While formal languages do not include syntax or semantics, they can be 
qualified to sets of working descriptions very easily through the intension 
definition [1 7] of sets, e.g. "the set of all working Al implementations." By 
contrast, a formal grammar requires a definition in the form of production 
rules that recognize or generate some language; this would potentially 
require enormous numbers of productions [18, 19, 20], and, in some cases, 
may be unknowable for practical reasons. This hints at the subtle 
relationship between conventional machine learning and the limits of 
narrow Al. In other words, without the relevant semantics at hand, one 
degenerates to the brute-force enumeration of countless permutations, all 
the while never having the capacity to effectively apply those rules outside 
of the context in which they were derived by rote calculation. This is 
because the problem of semantics is vastly more involved than the mere 
juxtaposition [21] of symbols, especially when those symbols have 
meaning. In the context of computational languages, this meaning comes 
in the form of functionality that must be captured in an implementation, a 
description that entails the semantics. 

4.2 Code-Data Duality 

Computers are primarily programmed through the transformation of 
human readable source code into descriptions that can be directly and 
indirectly executed by the machine. Compilation is the process of 
transforming that source code from one description language into that of 
another. This typically results in native machine code in the description 
language of some microprocessor or microcontroller. In other cases, the 
result of the compilation is generated in a description language that differs 
from the native description language of the machine and must be 
interpreted to be executed. Finally, source code may be interpreted directly, 
with or without a corresponding compilation process. 

The underlying hardware is always involved, it's just a question of how 
many layers of abstraction are between the given description and the 
machine's native description language, called the instruction set. It is 
possible to nest, encode, or embed descriptions within descriptions ad 
infinitum. The limits of this are determined by each description language 
and the resource constraints of the implementation. 

In all cases above, the descriptions are data being interpreted by some 
process, with that process being either another program or the machine 
itself. This is the duality between code and data, and it's the basis of self- 
modification in computing. This duality helps to unify the notion of 
interpreter and machine in the interpretation of information, which is what 
is being referred to when one discusses information exchange. In other 
words, information exchange is not possible without an interpreting 
process, and every interpreting process must have syntax and semantics as 
part of its construction, otherwise it would be incapable of signification. 

A description language with semantics that provide access, recognition, 
and generation of its syntax and semantics while being interpreted is called 
reflexive [22, 23]. However, this property is not required for self- 
modification in general. This is related to the class of self-interpreting 
languages that exhibit homoiconicity [24], which represent the 
implementation details of the description language in terms of the 



http://aisecurity.org/ref/self-modifying-systems/ 



2/19 




9/1/2015 



Self-Modifying Systems — Al Security 



structure of the language itself. This makes these languages trivially 
reflexive. In all cases, however, the limits of self-modification are up to the 
semantics of the interpreting process and not particular language features, 
which only make it more or less convenient to perform self-modification. 

4.3 Interpreters and Machines 

An interpreter is an implementation that evaluates descriptions written in a 
description language [25]. If in software, the interpreter would be either a 
program or embedded as part of another program. In hardware, it would 
be a microprocessor or microcontroller or, potentially, some customized 
integrated circuit or other hardware. The language that the interpreter 
hosts may be either the same or different from the one it was implemented 
in itself. 



Figure 4.1: Conceptual construction of interpreters as 
nested implementations of description languages. 



Host (Interpreter) 



Interpreter 
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Each box represents a single description out of possibly 
many descriptions entailed by the implementation that 
contains it. Every interpreter is a description in some descrip- 
tion language that also represents an implementation of a 
(possibly different) description language.This continues 
schematically towards the upper right diagonal ad Infinitum 
until reaching discrete physical descriptions. 

Every interpreter follows the construction above: a description in a host 
description language that implements another, possibly distinct, 
description language. The power of the interpreted language is always 
determined by the semantics of the implementation. This can be applied to 
nearly any context that permits a physical description of the system. 

In this book, all interpreters are generalized to a category or class under the 
above construction. This confers the advantage of a universal analysis that 
provides broad coverage of the relevant principles of self-modification. This 
generalization includes all of the types of interpreters that will be discussed 
in this section, both hardware and software alike, and should not be 
confused with the light-weight interpreters that are about to be discussed. 

Often, in the context of computer programming, interpreter typically refers 
to a specific sub-class of interpreters that utilize abstractions (or no 
abstractions) that are relatively close to the form or function of the 
description language they host. These interpreters either use some internal 
representation or perform direct execution as the language is parsed. In 
some cases, these interpreters will utilize what is known as byte-code [26], 
which is a compiled version of the human readable source that has been 
put into a description language that is more efficient for machine reading. 
This byte-code is usually not in the language of the host architecture that 
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runs the interpreter, and must be read by the interpreter itself. 

Just-in-Time (JIT) compilers combine the features of the light-weight 
interpreters above and that of compilation by allowing descriptions to be 
analyzed and transformed at runtime [27]. 

A virtual machine (VM) is a type of interpreter that goes a step further and 
implements, or, more correctly, emulates, the semantics of a particular 
computing architecture or platform [28], This creates an additional level of 
abstraction or indirection between the description language being hosted 
and the underlying architecture that implements the VM. The benefits of 
this is that the VM can trade relative performance for platform and 
hardware independence [29, 30]. It also admits the possibility of unique 
security features, as it is not compelled to execute every program or series 
of instructions that it encounters [31, 32, 33]. Hardware circuitry could be 
implemented to do the same [34, 36, 37, 38, 39, 40] but it may not be as 
flexible or dynamic as a VM. Ultimately, however, a VM is still an 
implementation, and any security at this level would still be self-security, 
and subject to fundamental vulnerabilities [41]. The VM concept primarily 
gains attention due to its ability to run descriptions on multiple 
architectures without recompilation. 

Virtualization takes the VM concept a step further by creating one or more 
VMs across computing systems of potentially different types of 
architectures [42], This can be used to split computational resources into 
smaller units or combine them to form a larger logical machine. 

Finally, a microcontroller or microprocessor is an interpreter implemented 
as hardware. It has an instruction set which exposes hardware semantics at 
the lowest levels. It is at this level that other interpreters are implemented, 
including VMs, which must have an associated native implementation 
portion to enable the emulation of the architecture. 

All interpreters have a set of advantages and disadvantages that are 
determined by their descriptions. There can be no general purpose 
description that is optimal for every type of algorithm or program. This 
truism is the result of the semantics of the implementation and the levels of 
abstraction between it and the discrete physics. Every layer of abstraction 
or analogy induces another level of interpretation that must be performed. 
As such, the most efficient descriptions are those that minimize these levels 
of indirection [43, 44, 45, 46, 47, 48], A digital computer is still an analog 
device in that it utilizes electrical states as representations for digital logic. 
Its storage mechanisms manipulate and interpret physical states which are 
analog representations constrained to specific ranges of interpretation. 

The optimal computer for a given problem is one in which there is as close 
to a one-to-one correspondence between the algorithm being 
implemented and the physical analogues used in its interpretation. This is 
perhaps why the human brain is such an efficient computational system, 
even without factoring in its ability to reconfigure itself as it undergoes 
learning. 

Further issues that confound any general purpose solution is that certain 
problems are recalcitrant to parallel processing [49, 50, 51]; some can 
benefit from the ordering of instructions and the prediction of branching in 
logic; others are benefited by cache in terms of memory and instructions, 
while others would need bigger memory bandwidth, as they do not benefit 
from cache as well as others due to their dynamics. The bottom line is that 
there are compromises in the design of any interpreter that are often 
decided by the context in which they will be used most frequently. This is 
another reason why it is almost always nonsensical to discuss in the 
absence of specific implementations. 

This is all related to self-modification in that the above limitations are 
exactly why a system would benefit from the ability to self-modify. These 
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are non-trivial problems that are unavoidable and recalcitrant. Take 
Amdahl's law, for example, which gives a relationship between the 
maximum speed-up that can be attained by adding more processors to a 
problem with even a fractional serial portion [52]. That is to say, any given 
problem can be represented by how much of it is necessarily serial in its 
execution. As mentioned above, not all problems benefit from parallel 
computation. 

Often, the most difficult part that professional software engineers and 
applied mathematicians have to struggle with is finding the optimal 
algorithms constrained by the relevant semantics of the description 
languages given to them. A professional graphics developer is a perfect 
example: they have cutting edge, specialized physical hardware 
descriptions which provide unique semantics, but these semantics are 
never as flexible as just specifying a physical implementation of the 
algorithm. This is because these systems have to fit within a manufacturing 
process and made general enough to serve a wide variety of applications; 
they are specialized, but not so specialized that they would only calculate a 
singular set of problems. 

In summary, an interpreter is always required for any description language 
to have effect. At its most fundamental, the physical world can be thought 
of as an interpreter with what we call reality being its descriptions. This is 
not to advertise a digital physics, as descriptions need not be represented 
digitally! It is to say that this concept of an interpreter is almost universal, 
depending on how willing one is to relax their conceptual boundaries. 

Every interpreter has to implement the semantics and recognize the syntax 
of the description language they host, with the power of that language 
being determined by the semantics of the implementation. The 
implementation's efficiency is determined partially by its syntax and 
semantics, plus the levels of abstraction between it and that of some 
discrete physical description. 

Every underlying physical interpreter involves a form of analog computing 
or representation, and the most efficient possible forms of computation 
would be those which are as close to a one-to-one correspondence with 
this as possible. This raises the fundamental trade-off between generality 
and specialization, in that the closer we arrive to this one-to-one 
correspondence, the more constrained the implementation becomes for a 
particular set of problems. This sets the bounds of the problem space for 
which self-modification is applied, as it seeks to have both generality and 
specialization through the self-modification process. 

4.4 Types of Self-Modification 

All systems capable of self-modification share a few basic operations: 
rewrite, replicate, and generate. These are either direct or indirect in 
operation. A description (implementation) is in either one of two states: 
online or offline. 

Offline descriptions are the conventional type of description that have been 
discussed so far about implementations. Recalling the time-like objects and 
processes from the previous chapter, offline descriptions entail potential 
states and operations and are not the physical description of an active 
implementation. They are an entailment of its potential. By contrast, an 
online description entails the relevant operational and physical information 
of an implication but only insofar as it is required to undergo self- 
modification. This may include state and environment information or 
anything else that would be required to complete self-modification under 
active operation. 

Direct self-modification is where the system is applying modifications to 
itself without external sources for aid. Indirect self-modification is for cases 
where the system utilizes external sources to self-modify, but with the 
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stipulation that it must have either created or initiated these sources. A 
supervised self-modification is still self-modification as long as it is able to 
operate directly or indirectly upon itself without further assistance. 

Intervention, cessation, or interruption of this process by a supervising 
process would not constitute aid unless it qualitatively alters the self- 
modification process beyond the starting or stopping of its execution. 

A summary of the types of self-modification: 

• Direct 

o Rewrite 

o Replicate-Rewrite-Replace 
o Generate-Replace 

• Indirect 

o Replicate-Rewrite-Replace 
o Generate-Retroactive Rewrite 

Rewrites involve partial or complete modification of the original 
description. There is no implied copying. Rewrites may apply to 
descriptions which are either online or offline. An offline description is 
simply a copy of a system's description which is no longer considered part 
of it. An online implementation that engages in rewrite self-modification 
must be capable of handling live edits of its description without generation 
or replication. This lack of copying is what distinguishes rewrite as a basic 
operation in self-modification from replication and generation. Thus, when 
rewrite occurs to an offline description it is necessarily an intermediate 
stage towards self-modification, as it will require an additional step to be 
considered part of the original system once more. 

Replication is the unmodified copying of a system's description. It is an 
intermediate step towards self-modification. This replication is capable of 
occurring within a system without instantiating the copied description. It 
may be a temporary working copy or part of the process of self- 
modification. Replication does not imply an online description, but it does 
imply replacement if it is to be considered part of any self-modifying 
process. 

Generation can be thought of as a combined replicate and rewrite process, 
but is distinct from both as it is the direct construction of a description 
from a process. This is to be contrasted with the replication, rewrite, and 
replace method of self-modification, which involves the modification of a 
copy. The defining characteristic of generation is that it involves 
computation, and, as such, can be generalized as a kind of compression, in 
which the generative system has an effective "copy" of all the possible 
permutations it can potentially create in an ultra-compact representation. 

Like replication, generation implies an eventual replace to be considered 
self-modification. This is because generation creates a separate description 
and does not modify the original system (such a case would be a rewrite). 

Retroactive replacement is for the complex case where a system is, for 
some reason, incapable of direct self-modification. As a result, it generates 
or replicates a description, which may or may not need to be modified 
further, and then utilizes that as a means of self-modification. 

Finally, there needs to be a discussion regarding iteration. Why would this 
not be considered self-modification? The reason is that it does not imply a 
change in the identity of the originating process and that it does not serve 
the same function or purpose of self-modification, which is to break 
through description language barriers (to be discussed ahead). 

4.5 Reconfigurable Hardware 

Application-specific integrated circuits (ASIC) are customized integrated 
circuits designed for a specific functionality and purpose [53]. They range 
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from partial to fully custom designs that can vary greatly in performance 
and cost. The benefit of ASICs are their high performance and lower 
marginal cost at high production volumes [54]. The drawback is that they 
can not be reconfigured once manufactured. 

By contrast, and of relevance to self-modification, is the field 
programmable gate array (FPGA), which uses an array of programmable 
logic blocks that allow significant changes to a circuit design, and can be 
re-configured and reused for many applications [55, 56, 57]. Limitations 
include only partial update during operation [58, 59], a significantly more 
complex reprogramming process than software [60], and volatile storage of 
the configuration [61]. The latter issue of volatility can be overcome with 
system-on-a-chip designs that include common integrated circuit (1C) 
components as part of the FPGA itself [62, 63, 64], This hardware is 
commonly used to prototype and verify ASIC designs [65, 66, 67] and can 
potentially be more cost effective than an ASIC, at low volumes, even for 
production use [68], 

Firmware is defined as the combination of integrated circuits (ICs) with 
non-volatile memory to allow logic to be represented as software directly 
in the hardware [69]. This provides some flexibility and modularity to the 
hardware but not to the same extent as an ASIC or FPGA. This is because 
the stored software is still forced to operate on a specific circuit and data- 
path configuration, and it can not be altered unless combined with one of 
the above configurable devices. 

At the time of this writing, there exists no configurable hardware 
technology that compares with the freedom and ease of self-modification 
available to software. The trade-off, however, is run-time efficiency. This 
means that while software is capable of virtually unrestricted self- 
modification in terms of algorithmic creativity, it comes at the cost of being 
limited (in efficiency) by the semantics of the hardware that serves it. This 
could be limiting in cases where solutions benefit significantly from 
parallelization, but it is important to note that this does not fundamentally 
prevent the ability to run these implementations at a less optimal pace. A 
slow strong Al could potentially be significantly more effective than even 
the brightest human. 

Reconfigurable hardware, including FPGA, firmware, and embedded 
systems also have their own set of unique security challenges [70, 71, 72, 

73, 74, 75, 76, 77, 78, 79]. It is because of this, and the above benefits of 
software, that the focus of this book is not on hardware, but software. The 
reasons are clear: no current hardware technology has the ability to 
efficiently and effectively self-replicate or modify at sufficient rates. 

Software does, and, with the increasing ubiquity in connectivity, it has the 
means to spread very rapidly. Thus, the rest of this chapter will focus on the 
software aspects, which follows more closely inline with the foundations 
presented in the previous chapter on Al implementations. 

This does not mean that we should ignore the physical parts of 
implementations or the vulnerabilities in hardware. It is only to say that, in 
the context of self-modification, the path of least resistance will be 
software. That this is the route most likely to appear first and the one to be 
taken most easily by malicious users. It will also be the most difficult to 
restrict, enforce, and contain; properties that make it a prime medium for 
misuse. 

4.6 Purpose and Function of Self-Modification 

Descriptions are incapable of transcending their respective description 
languages without transformation. This is the fundamental barrier that self- 
modification seeks to overcome. If we want an optimized set of semantics 
or a more efficient syntax, we have to have different descriptions. To get 
from one description to another requires an explicit process. Self- 
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modification occurs when this transformation takes place within a single 
identity and the entity at the source and destination are (eventually) the 
same. 

As was defined above, the description languages discussed in this book 
pertain to working, well-formed, possible descriptions, along with their 
relevant syntax and semantics. As a result, an interpreter can be seen as an 
implementation of a description language. In such a case, we are referring 
not to the description of the interpreter but to that of the possible 
programs of the description language it entails. This brings us to the actual 
focus of this section: 

Figure 4.2: Language barriers as fundamental limits in 

self-modifying systems. 



Interpreter 




Any description is contained within the description language 
implemented by an interpreter unless there are enabling 
semantics to transcend it. This applies universally to any 
physical description that can undergo (self-)modification. 

In the figure above, each box represents a description. The description 
undergoing self-modification is the smallest, innermost box in the lower 
left. It represents only a single instance out of many possible descriptions 
entailed by the interpreter. What this shows us is that the self-modifying 
implementation depicted is capable of attaining any possible description in 
the interpreter's set of possible programs. This is a trivial form of self- 
modification, as it is still locked within the description language of the 
interpreter, and of the description language that implements that 
interpreter, ad infinitum. In practice this is never infinite, being bound by 
practical limits and real-time demands. A discrete physical description is 
attained relatively quickly, semantics permitting. 

This first trivial form of self-modification illustrates the line of demarcation 
between a self-hosting interpreter and a meta-circular interpreter. Typically, 
a self-hosting interpreter implements its description language semantics 
directly. This is neither unusual nor interesting as a point itself; however, 
when contrasted with meta-circular interpreters, it becomes a limitation. 
This is because the description is locked within a self-interpreter's 
framework. Meta-circular interpreters (only partially) overcome this by 
exposing the underlying interpreter that hosts it [80, 81]. This makes lower 
level semantics part of the semantics of the description language, and, as a 
result, effectively makes the interpreter transparent and reflexive. This 
allows for the meta-circular interpreter to implement new description 
languages by building on the primitives of its underlying host semantics. It 
may seem an ideal candidate for self-modification, but this too is still 
limited when compared with the next stage. 

The most powerful form of self-modification is where the interpreter can be 
bypassed entirely for the next stage of interpretation: 
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Figure 4.3: Self-modification transcending description 
language barriers. 




The diagonal represents what the self-modifying description 
(lower left) is able to achieve with enabling semantics. 

Bypassing the interpreter LI, it is free to penetrate levels of 
abstraction, including L2 and beyond, until a semantic barrier 
or a discrete physical description is reached. 

In the above figure, the self-modifying program is breaking through its 
original interpreter and accessing the interpreters at each successive stage. 
Unlike the light-weight interpreters previously described, it has the ability 
to completely escape each description language boundary. That is, it is 
activating and bypassing levels of abstraction in terms of interpreters, 
which are implementations of description languages. It can do this until it 
reaches the level of the discrete physics, where, hypothetically, it would 
only be limited by the material and energy resources available to it through 
some form of physical reconfiguration. Note that this is beyond the level of 
reconfigurable circuits, and would require nanotechnology or an equivalent 
process that can manipulate the physical description of the computational 
substrate. This diagonal escape from each description language is possible 
as long as the semantics exist to access the next lower level at that stage in 
the process. 

Figure 4.4: Self-modification transcending description 
language barriers through peripheral or network access. 




This process could be achieved through any means which 
enables the self-modification process.This would still be 
limited by enabling semantics and the required knowledge 
of the destination description languages and interpreters. 

If a limitation is reached where there are no longer any semantics that 
implement next lower-level access then it is possible for the self-modifying 
system to escape that meta-stable state by accessing network or 
peripherals. This is what is diagrammatically occurring in the above figure. 
In this case, it could be Internet access to a remote machine with a 
description language that permits access to semantics that allow for self- 
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modification to penetrate to the next lower-level. 

It is important to understand that the semantics for self-modification are 
non-trivial. Semantics have to exist for the process of seif-modification to 
break through these barriers. The process is not magic. It simply does not 
have an ability to penetrate a description language barrier without the 
relevant methods at its disposal. 

It is trivial to construct such curtailed worlds. It is done through an 
interpreter that simply lacks the semantics that would allow descriptions to 
manipulate APIs, I/O, device drivers, and so forth. Any programs under such 
a description language would be shut into a self-contained domain, a world 
with no possible escape without external aid. This is in stark contrast to one 
in which the semantics exist but have been guarded through various 
security measures. 

The lack of enabling semantics should not be taken as an inability for such 
a system to communicate outside of the locked in world, which might be 
possible through side-channel attacks, such as timing analysis [82]. This, 
however, could be mitigated by implementing a real-time constraint on 
computation, such that the host interpreter is computing at a uniform or 
sufficiently uncorrelated rate with respect to program complexity. 

The above methods describe various levels of sandboxing, and must not be 
taken as an argument for self-security. A sandboxed system can be 
overcome through external means, flaws in the implementation, and soft- 
errors regardless of the presence or absence of semantics and no matter 
how well guarded. The reason this discussion is made here regarding 
enabling semantics is to draw parallels between the existence and non- 
existence of doorways for self-modification. 

As discussed above, the greater the distance between an implementation 
and the analogue representation in the discrete physics, the poorer the 
efficiency. As such, self-modification is a means of increasing the efficiency 
of a system by accessing either better semantics or by approaching a closer 
correspondence to an optimal state of computation for the problem. While 
this may sound abysmally utilitarian, it can be the difference between a 
tractable and intractable solution; the actual ranges for decent real-time 
performance are actually small, and it can be extremely difficult to get 
complex computation down into such ranges, especially on general 
purpose hardware. The challenges are not always in thinking up a method 
to solve a problem, but in making it fast on available hardware. 

4.7 Metamorphic Strong Al 

A metamorphic program [83, 94] is capable of recognizing and 
manipulating its own description language(s). The parenthetic plural is due 
to the fact that a program description may have more than one 
representation in the form of compiled code; a description in the human- 
readable source form, which may or may not be available to the program, 
and the native machine code representation. It is the latter that is most 
pressing with respect to metamorphic programs, as it automatically grants 
the ability to recognize any other program that has been compiled for the 
same architecture. 

These programs are distinct from self-hosted interpreters in that their 
primary purpose is not to facilitate or implement program descriptions but 
to implement a particular set of program features which may change over 
time. Such programs utilize self-modification as a means, not an end, for 
enhancing these features or to discover new ones entirely. 

The advantage of viewing strong Al as metamorphic software is that it 
simplifies the analysis. From a security standpoint, we can focus on the 
description, how and when it changes, and what other descriptions it 
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modifies. We do not necessarily have to understand the description or the 
program behavior it implements. This is tremendously powerful, as we can 
recognize and classify strong Al by their descriptions and the description 
languages they are able to recognize. This could be used to create a 
hierarchy of increasingly complex autonomous systems that are graded 
based on their ability to manipulate these description languages or acquire 
new ones. 

If we admit a representation where a strong Al description or 
implementation includes what it learns as part of its total description, then 
we have a basis for dovetailing all of current studies in artificial intelligence 
into the framework of self-modifying systems. That is, self-modification 
need not be restricted to what is typically thought of as machine op-codes 
and low-level programming constructs. Recall that description languages 
may be induced where any consistent structure can be applied. That they 
can then be connected with the rest of computer science and mathematics 
is what lends to their beauty as a unifying basis. Thus, that an 
implementation is modifying itself does not necessarily imply that it is 
altering its architecture; it could be modifying its knowledge and memories, 
or, in a more complex case, unavoidably altering its computational 
substrata because there is no longer a distinction between discrete units of 
computation and units of storage. 

Machine learning comes into play at the intersection between the 
acquisition of new description languages and the manipulation of the 
descriptions in self-modifying implementations. This generalizes the notion 
of machine learning to that of the adaptation and effective manipulation of 
descriptions in some set of description languages, with the breadth and 
quality of that set defining a measure of its raw intellectual capacity. This 
works so long as one is willing to admit a flexible interpretation of 
description languages; they can be applied to virtually any physical process 
or system, even those that are under-specified, such as the micro and 
macro features of non-verbal communication in humans, or the particular 
way a set of servos must apply forces for some specific robotic instrument, 
all of which may vary from implementation to implementation due to 
subtle imperfections and environmental effects. Each is a description 
language in that it specifies a code, a way of transforming representations, 
and, with that comes consistent structure, lest it would be 
incomprehensible or unpredictable. 

This is, however, non-trivial, as the semantics for any new description 
languages would need to be known in advance or inferred from an existing 
set of implementation semantics. Without this, the implementation would 
have to fall back to associative approaches that rely on mining data and 
constructing very high-dimensional representations of what could have 
potentially been vastly simpler structures. A case in point would be an 
attempt at mining all the ways one could signal a greeting, with no 
semantics for greetings available. This would result in brute enumeration of 
any possible combination in auditory, visual, and other modalities. It would, 
of course, collapse to descriptions, with the modalities simply being 
description languages. The system would then have to construct n-grams 
representing chains of presumed independent events that build a massive 
state space of possible behaviors based on where one is without regard to 
where one has been [84, 85, 93]. 

The Markov property [86] just described is essentially the default of any 
system that lacks semantic capacity; it only has incidence at its disposal. 

This is the degenerate case of any narrow Al system, as it necessarily lacks 
the semantic faculties that a strong Al must have. This is not a weakness or 
fault in the foundations but a reflection in the choice of philosophy or 
epistemology in engineering practice. It's a misreading or misapplication of 
how knowledge is acquired by ignoring the vast compression that 
signification represents. That is to say, signification is merely an 
externalization of associated semantics. This is not to be confused with tacit 
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versus explicit knowledge [87]. There is a gulf between semantics and even 
tacit knowledge, with the semantics of an implementation presupposing 
any possible knowledge representation, be it tacit or explicit. This is 
completely overlooked in conventional learning algorithm construction, not 
even on the table for consideration; it simply doesn't exist in the 
conceptual space of the relevant literature. 

How does this apply to self-modifying systems? In that it sets up a barrier 
that a metamorphic program may possibly overcome due to the fact that 
its ability to self-interpret is independent of its ability to be executed. This 
allows the possibility of acquiring the use of description language 
semantics that were not previously available to the language that created 
it. Such a feat is not directly possible through self-hosting interpreters, even 
meta-circular ones, as there is always a footprint or underlying 
implementation in the host description language which is required to 
recognize the structure of the description language being implemented. 
This is not the case with metamorphic programs, as they are capable of 
transcending architectures and instruction sets. Such programs would have 
the capacity, through trial-and-error, or, the relation of existing semantics, 
to test and probe for semantics in the new description languages in which 
they rewrite themselves. This process would necessarily be error-prone and 
slow, no matter how "intelligent" a system (slow, relative to direct 
application of existing knowledge). 

The analysis of metamorphic programs thus admits a convergence with 
bioinformatics and systems biology, in that we can directly interpret these 
implementations through the same techniques used for detecting 
mutations, insertions, and deletions in genetic sequences. All we must do 
to begin is admit is that there are multiple description languages. We 
would use generalized algorithms, methods, and a kind of multi- 
dimensional cladistics to recognize the behavior of these systems across 
description language boundaries. 

Moving away from the theoretical now and towards the practical, we arrive 
at the operation of metamorphic programs. Traditionally, this has been to 
evade detection through masking their unique fingerprint or signature, and 
to confuse heuristics used by anti-malware tools [88], This is a battle 
between metamorphic malware and the countermeasures used to detect 
them, with the countermeasures on the losing side [89]. In the end, the only 
comparable defense will be to instrument strong Al that can anticipate and 
adapt to other metamorphic strong Al. Crucally, all strong Al will essentially 
be metamorphic. 

To quickly recap: 

• Self-modification has the potential to overcome description language 
barriers and affords opportunities for the potential acquisition of new 
semantics. 

• Manipulation and acquisition of description languages determine the 
threat model of metamorphic Al. 

• The acquisition of semantics is non-trivial, and, in its absence, 
degenerates knowledge acquisition to high-dimensional rote 
grammars from brute-force enumeration. 

• Breaking the meta-stability of self-modification is not done through 
magic. It requires enabling semantics to shatter the language barriers 
that keep it there, e.g. the ability to perform rewrite, replicate, or 
generate operations. 

• The execution of metamorphic programs is independent of their 
ability to self-interpret. 

• Systems biology and cladistics can be used to categorize and classify 
metamorphic descriptions, which could lead to better understanding 
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and actionable knowledge. 

So far, this section has primarily focused on the positive or neutral effects 
of self-modification, but it is in the threats that the metamorphic 
perspective comes into full perspective. Consider programs which have the 
ability to rewrite restricted Al implementations, turning them into fully or 
partially unrestricted versions or restructuring them entirely to carry out 
malicious intent. Instead of malware that disrupts a specialized system or 
network, you could have adaptive or generalized strong Al malware that 
spreads through our global communications infrastructure, destroying or 
subverting existing implementations that have become compromised. This 
is an eventuality that this analysis can help to at least understand and 
anticipate. 

The above statements come with a serious qualifier: strong Al is not going 
to spontaneously develop a persona and a sense of survival ex nihilo, then 
suddenly seek to overcome our civilization. Rather, it will be mere human 
beings who will craft these implementations and then release malicious 
versions of strong Al into the world. It could be a person or group doing it 
for the "lulz" (enjoyment in exercising the power to destroy simply because 
they can) [90], or it could be a government or non-state actor. All of these 
will be serious threats to cybersecurity and to the societies that are 
impacted. This is not some nebulous dollar figure attached to lost 
productivity or hampering of business as usual. Metamorphic strong Al 
malware will be beyond human-level malware, and will have the potential 
capacity to act as a local beligerent to the system beyond mere replication 
and disruption; it provides intelligence as a payload, and is the ultimate 
form of cybersecurity warfare. They will be virtual agents behind the lines, 
capable of exploiting and adapting vulnerabilities that would be impossible 
to detect remotely. The sword cuts both ways, however, as it is also going 
to be the optimal defense. 

Metamorphic strong Al would be capable of analyzing existing machine 
code for vulnerabilities that would be beyond the ability for human experts 
to reasonably parse and analyze at that level, even with directly access to 
the information. Used defensively, these programs would be benign 
versions of the same destructive metamorphic software described above; 
instead of triggering a replicate or rewrite phase, it would report that 
information for human intervention. And it is at this level that we will first 
see strong Al instrumented and used on a wide scale in the information 
and security sectors. This is perhaps the closest reason for the name of this 
book: to become Al secure is to utilize the most effective strong Al 
implementation to seek out and eliminate vulnerabilities that only they 
could reliably detect and overcome. One's system may be secure, but it 
may not be Al secure. This cleanly entails the essence of the threat model. 
We won't be dealing with just human minds, but automated processes and 
cognitive implementations that are effective at levels that are beyond our 
best and brightest. In such a case, a metamorphic analysis is central in a 
warfare based on quickly changing descriptions across many architectures. 

To employ these artificial minds in our defense, however, we must 
understand the technical, philosophical, and ethical implications of machine 
consciousness. 
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